Supplementary Material to Robust Structured Estimation with Single-Index Models

In this supplementary material, we present the deferred proofs of the results in the main paper. 1. Proof of Claim 1 Statement of Claim 1: Suppose that each element xi of x is sampled i.i.d. from Rademacher distribution, i.e., P(xi = 1) = P(xi = −1) = 0.5. Under model (3) with noise ε = 0, there exists a θ̄ ∈ Sp−1 together with a monotone f̄ , such that supp(θ̄) = supp(θ∗) and yi = f̄(⟨θ̄,xi⟩) for data {(xi, yi)}i=1 with arbitrarily large sample size n, while ∥θ̄ − θ∥2 > δ for some constant δ. Proof: In the noiseless setting with unknown f∗, provided that S , supp(θ∗) is given and |S| = s, the estimation of θ∗ is simplified as Find θS ∈ Ss−1 s.t. sign ( ⟨θS ,xiS − xjS⟩ ) = sign(yi − yj), (S.1) ∀ 1 ≤ i < j ≤ n , any of whose solution θ can be true θ∗ on the premise that no other information is available, since there always exists a monotone f satisfying f(⟨θ,xi⟩) = yi. Given the distribution of x, xiS − xjS only has 3 s possibilities even if n → +∞. We denote the feasible set of (S.1) by C, which is basically an intersection of Ss−1 and at most min{n(n− 1), 3} halfspaces (or hyperplanes if yi = yj). Depending on the 3 different values of each sign(yi − yj), this feasible set C has at most 3min{n(n−1),3p} possibilities, which is finite, and the union of them should be Ss−1. When s ≥ 2 and the constant δ is small enough, we can always find a C, in which there exist two different points away by δ. Specify them as θ∗S and θ̄S respectively, and Department of Computer Science & Engineering, University of Minnesota-Twin Cities, Minnesota, USA. Correspondence to: Sheng Chen , Arindam Banerjee . Proceedings of the 34 th International Conference on Machine Learning, Sydney, Australia, 2017. JMLR: W&CP.; Copyright 2017 by the author(s). we are unable to distinguish between them, as both can be solution to (S.1) for any samples. 2. Proof of Lemma 1 Statement of Lemma 1: Suppose the distribution of y in model (1) depends on x through ⟨θ∗,x⟩ and we define accordingly bi (z1, . . . , zm;θ ∗) = (S.2) E [qi (y1, . . . , ym) |⟨θ,x1⟩ = z1, . . . , ⟨θ,xm⟩ = zm] , With x being standard Gaussian N (0, I), u defined in (4) satisfies E [u ((x1, y1), . . . , (xm, ym))] = βθ∗ , (S.3) where β = ∑m i=1 E[bi (g1, . . . , gm;θ) · gi], and g1, . . . , gm are i.i.d. standard Gaussian. Proof: Let θ⊥ be any vector orthogonal to θ∗. For convenience, we use the shorthand notation u for u ((x1, y1), . . . , (xm, ym)). Then we have